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•ABSTRACT 

This annotated bibliography lists instructional . 
development resources relevant tc the Interservice Procedures for 
Instructional Systems Development Model (TSDl , a standardized modti 
providing for the assessment of ^raining needs; the design, 
development, and implementation of instruction; and the assessment flt 
instructional quality. Po.llowind a literature search, relevant 
documents were classified according to the 19 block ISC model, and 
summaries were written to identify documents on authoring aids, 
procedures, or techniques. The nurcose of each block in this model is 
defined, and documents are listed alphabetically withio xblccks. A 
status ^ect ion for each bloc* indicates the availability of authoring 
aids sufficient to guide ar. individual throuqh all activities 
specified by the block, as well as the availability of relevant 
procedures and techniques that could be developed ictc authoring 
aids. Directions for future research, based on the lack of authoring 
aids available, are identified. <RA0* 
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•BRIEF 



Requirement; 

J 

To determine the state-of-the-art in instructional systems develop- 
ment and to specify critical research gaps* 



Procedure: 

A review of the instructional development literature was conducted* 
An annotated bibliography was produced which organised references accord- 
ing to the steps of the Interservice Procedures for Instructional Systems 
Development Model (ISO, TRADOC Pamphlet 350-30)* Each reference was 
classified as <1) an authoring aid, i*e* , a "how-to-do-it" manual which 
directly guides the author through an activity identified by the 1SD 
model,. (2) a procedure, i*e*, a listing of the activities Involved in 
TSD or ( ^) a technique, i,e*, a methodology for aQcomplishlng an activ- 
ity but # lackdng sufficient detail to be a procedure* 



Findings; 

(1) In Phase 1 - ANALYZE , authoring aids are available for Select 
Tasks/Functions? ^Construct Job Performance ^feasuresj Analyze Existing 
Courses; and Select Instructional Setting* Authoring aids are unavail- 
able for Analyze Job* 

(2) In Phase 1 1- DESIGN , authoring aids are available f .>r Develop 
objectives and Develop Tests* Authoring #ids are unavailable for De- 
scribe Entry Behavior and Determine Sequence and structure* 

(3) In Phase III -DEVELOP , authoring aids are available for all 
activities involving print as a medium* However, authoring aids are 
unavailable for computer-based instructional materials. 



(4) In Phase I V- IMPL^My ^authori n g aids are unavailable. 

(5) In Phase V - CONTROL , authoring aids are unavailable* 

* ft 



Vi1 



* 



Utilization of FVdi^gst 

The identification of *4ds, procedures, or techniques will be of 

immediate use to those involved in instructional systems development. 

In addition, these findings will form one basis for a future' research ♦ 
and development plan. 
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AN ANNOTATED BIBLIOGRAPHY FOR INSTRUCTIONAL 
SYSTEMS DEVELOPMENT* 



INTRODUCTION 

The need for the Instructional Systems Development (ISO) was iden* 
tified m fi*»li tests of tntiivi dualized instructional training systems* 
It was difficult to bring individuals who used these systems to the 
level of competence needed to produce good curriculum materials in a 
reasonable time at an acceptable cost* Similar difficulties have been 
noted with other systems employing computer-based instruction* This 
problem is especially acute in the Army because of the high turnover* 
of personnel* Further , since future Army training and education sys- 
tems will be tnii vi dualized, the need for this technology is likely 
to intensify* 

Currently the instructional systems development process is time con- 
sumtng, it is not unusual to find that it takes one hundred or more 
hours to ieveiop and evaluate one instructional hour of self-paced mate- 
rial* For example, orlansky (1979) provides evidence for variability and 
lenqth of time among authors to produce instructional material*. His data # 
which summarise all available DoD studies in computer-based instruction, 
indicates that authors vary by a factor of 10 among themselves. In addi- 
tion, several sources {e*g., Avner, 1979) have indicated that the training 
process takes at least six months before authors can be expected to pro- 
duce useful materials and a year before full production* 

The Army*s first attempt at instructional systems development was 
called systems engineering (United States Continental Army Command Regu- 
lation JS0- 100-1, February 4968)* The most recent effort of the Army 
to provide standardised procedures for the development and conduct of 
traininq is the inter service Procedures for Instructional Systems Devel- 
opment Model (ISO, 1975)* The model provides for the assessment of train- 
ing needs; the design, development and implementation of instruction and 
the assessment of instructional quality through a five phase process (see 
Figure D* The first phase is ANALYZE , which provides guidance on task 
analysis, the selection of tasks to be trained, the development of measures 
of job performance and the selection'* of t.he appropriate environment for 
training* DESIGN is the second phase which establishes the objectives, 
test items, and equence of the course* The entry behavior or skills 
the trainee arrives with are also noted during this phase* The third 
phase is DEVELOPMENT during which the instructional materials are created* 
Existing materials are examine/1 so that new course materials are devised 
only when appropriate others do not exist* A plan specifying all activi- 
ties of the learner is made as well as a plan for p\lot testing the newly 
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riGURF 1. Interservice Procedures for Instructional 
Systems Development Model 



developed materials, Durinq IMPLEMENTATION, Phase IV, the plan of Phase 
lit is activated with particular attention to the personnel needed to 
accomplish the plan* The quality of the instruction is assessed during 
Phase V CONTROL in tei ms of new skills acquired by the trainee and the 
fulfilled needs of the Command* As is shown in Figure 1, the five phases 
(identified by Poman numerals) have been elaborated upon in 19 blocks, 
(Arabic numbers ), each with its own inputs, outputs, and feedback loop 
following the requirements of a systems engineering approach* A brie! 
example on notation is Block JI.1 which refers to the second phase-DEStGN 
and first block— Develop Otpectives* 

The renamler of this report will briefly discuss a problem observed 
^with the implementation of the ISD model, provide an annotated biblio- 
graphy addressing the problem and pose directions for further research 
and development* 

PURPOSE 

The activities of military instructs developers have been speci- 
fied by a TRADOC Peculation (Pamphlet 3U0-3G) which is formally known as 

Interservice Procedures for Instructional Systems D e velopment Model 
(ISO)- This document provides "procedures* 1 to be folic ^ed when devising 
a course of instruction* These procedures reflect the state-of-the-art 
when the model was devised* Unfortunately little guidance beyond "what 
needs to b*» done" is provided by the ISD model* 
V. 

Tho present- report was designed to review instructional development 
resources in 10 effort to'provide more proceduralized guidance to indi- 
viduals tasked with implementing the ISD model* These individuals are 
referred to as course authors* By definition authoring materials may 
take, the form of research studies, textbooks, handbooks, computer programs, 
audiovisual sources, or technical manuals as long as the intention of 
the resource is to guide the analysis, design, development, implementa- 
tion, or control of instruction* Several topics are outside the boundary 
conditions of this report* These include research and development related 
to oo*t-ef*f>ctivew*ss *e*g*, Seidel and Wagner, 1979) and artificial in- 
telligence applications in instruction (e*g. , Gentner, 1979) . 

METHOD 

The present research was intended>to update and supplement a pre- 
vious literature review and annotated bibliography of authoring mate- 
rials which was prepared and updated by Logan (1977, 1979). Many of 
Logan *s conventions were adopted so that this report could be used in 
coniunotion with his* , However, this document includes the most cogent 
of Loam's references so that it is complete in itself as well. The 
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bibliography was f orme i over the course of four activities; search, 
classify, summary*, and assess* 

literature search, beginning with documents published in 1960, 
was conducted manually throuqh the library of the U*S* Army Research 
Institute for the Behavioral and Social Sviencen, Psychological Ab- 
stracts ani abstracts from meetings of relevant professional organiza- 
tions* Computer-based data searches were made with the Defense Documenta 
tion Center (DDC), the National Technical Information Center (NTI5), and 
the Educational Resources Information Center {ERIC) * Some of the de- 
scriptors used in the search activities were: 

* 

Audiovisual Communications Media 
Behavioral Objectives 
Computer-Ass i sted I nst r uct ion 
>>mputer-Mana,jed instruction 
E ducat ion 

Educational Program Evaluation 

In iividualized Instruction 

Instructional Materials 

Job Analysis 

Systems Analysis 

Task analysis 

Test Construction 

Testing 

Training 

!)n»wnts were also made available through personal contacts with re- * 
soat otters and* practitioners knowledgeable in instructional systems devel- 
opment* Copies of relevant documents were made on paper or microfiche 
ani manually stored* 

All documents were then classified according to the 19 block scheme 
indicated by the ISD model (see Figure 1)* In some cases a document ap- 
plied to more than one block, 1 In those cases it was annotated for each 
hlork* * 

N^xt, a short summary waJ written after a thorough reading of each 
document* If a document was summarized, both authors felt it was of 
vtlue. The purpose of the summary statement was to identify the docu- 
:^>nt .is^n authoring aid, procedure, or technique* An authoring aid 

defined as a "how to do it* manual which directly guides the author 
through an activity identified by the ISD model* A procedure was de- 
fined as a listing of the activities involved in a block of the ISD 
model. A technique was defined as a methodology for accomplishing an 
*eti/ity identified by the ISD model but lacking sufficient detail to 
bf> ,i Procedure. Finally, the status of authoring aids and procedures 
i iiiird* wa^ made on a block by block basis. 
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In the RESULTS section which follows, the annotated bibliography ,is 
presented with a summary of the purpose of each blocfc* The references are 
organized by block and then alphabetically within block* A letter pre- 
ceding each reference indicates whether it is an authoring aid (A), pro- 
cedure (P) , or technique (T) « A reference numbered A 2*4.8 identifies 
an authoring aid for Phase II-DESIGN, Block)4-Detormine Sequence and 
Stnicture-^hxch is the eighth source* The "notation in Arabic numbers 
identifying the source is consistent with Logan {1979), 

A Status section for each block indicates the availability -of author* 
mq aid* sufficient to guide an individual through all activities speci- 
fied by the block as well as the availability of relevant procedures and' 
techniques which could be, developed into authoring aids* A ^figure in- 
dicating the availability of authoring aids* procedures, and techniques 
by^ase and block follows, using the A, T** and P notatiorr* established 
earlier* , 4 

In some cases the status of a block may have indicated procedures but 
no techniques. It can be assumed that if a procedure exists a technique 
was available prior to its development* Educational Research Information 
Canter or National Technical Information Service numbers were noted when 
available* 

It should be noted once again that the main intention of £he anno- 
tated bibliography was to identify authoring aids. The inclusion ot pro- 
cedures and techniques was intended to direct developmental efforts 
toward authoring aids* 
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• . • RESULTS 

v 

AN ANNOTATED BIBLIOGRAPHY 

Purpose of Block 1*1 Analyze Job / 
To determine which activities compose a job, what level of perform- 
ance is adequate, under what conditions the job is done and in what 
order* , 
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P 1.U2 Christal, R. E. The United States Air Force Occupational 
Research Project ( AFHRL-TR-73— 75) . Lackland Air Force 
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cupational Research Division, January 1974* (AD 774 574) 
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pational Research Project which began in 1958 . The project has pro- 
duced the Comprehensive Occupational Data Analysis Programs package 
(CODAP) , CODAP is a computer-based system which inputs, reduces, and 
analyzes raw data from job inventories. It also generates reports 
identifying individual and group job characteristics and between-job 
differences* It is currently used by all of the Armed Forces. 
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T 1.1.3 Control Data Corporation. Instructional task analysis 

(Pub* No. 76360712). Minneapolis, MN; Control Data 
Data Corporation, 1978. 

This document reviews three technique? for task analysis — ( 1) job task 
analysis, (2) information-processing task analysis, and (3) hierarchi- 
cal task analysis. 

P 1.1*4 Folley, J . D. Jr. Development of an improved method 

task analysis (Tech. Report). Villencia, PA: Ap- 
plied Science Associates, June 1964. (AD 445 869) 

Folley provides a detailed procedure for task analysis indicating what 

information is .needed and how it should be obtained and recorded. 

T 1.1.5 Mccormick, E. J., De Nisi, A. S* Shaw, J. B. The use 
of the position analysis questionnaire (PAfl) for es- 
tablishing the job component validity of tests {Report 
No. 5). West Lafayette, IN: Purdue University, 1977. 
(AD A042 270) 
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A research program is discussed which used the PAQ, a structured job 
analysis technique to establish "job component validity" of tests. The 
PAQ is a complicated questionnaire which enables the respondent to 
analyse a 30b in relation to 187 job elements describing job related 
.behaviors. 

P 1.1.6 Merrill, P. F. Task > analysis — An information process- 
ing approach. National Society jtor Performance and 
Instruction Journal , 1976. 15(2i-7-11. 

This article reviews Gagne's hierarchical task analysis procedure and 
presents an alternative approach — information processing analysis. 
Merrill sugqests that two skills have an hierarchical relationship if 
learning one skill is prerequisite to learniig^the other. An informa- 
tion processing relationship may be observed if "the output^ of one 
operation are required as the inputs for another operation." 

T 1.1.7 Montemerlo, M. D. &. Eddowe's, E. The judgmental nature 
of task analysis. Proceedings of the Human Factors 
Society Annual Meeting . Detroit, 1978. 

This paper examines the state of the art of task analysis. Montemerlo 
and En-Mowes sugqest that "task analysis" defies task analysis. 'They 
us* the term "proceduralizable" to characterize those tasks which can be 
described. Task analysis which is largely judgmental does not lend 
itself to proce<iuralization. Their argument is supported by seven fac- 
tors which indicate that task analysis is an artistic, creative, syner- 
gist ic, multi-purpose, problem solving, global, interpersonal, political, 
and cognitive task. They conclude that task analysis cannot be reduced 
to one specific procedure. 

P 1.1 .'M Powers, T. E. Selecting presentation modes according to 
p ersonnel characteristics of users and the nature of 
job tasks, Part I job tasks . Baltimore,. MD: Uni- 
versity of Maryland, Baltimore County, July-Dec. 1977. 
(AD A038 511) 

In this volume. Powers examines the usefulness of ten job task cate- 
gories in an effort to provide a unif rm procedure for classifying any 
lob task, seven categories were found to be highly applicable to many 
technical job tasks. Other volumes examine a task analysis, technique 
which distinguishes the way technical information is presented accord- 
ing to personnel aptitude and category of job task. 

T 1.1.9 Task Inventory Exchange. Brochure . Columbus, OH:' The 
Center for Vocational Education, » The Ohio State 
University, 1975. 
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The Task Inventory Exchange (TIE) is a national clearinghouse for occu- 
pational task 'inventories which are lists of job tasks* The goal of 

th<» TtR is to increase the sharing of task inventories in ord&r to 
reduce duplication. 

P 1.1*10 Traminq and Doctrine Command. Job and task analysis 
regulation (TRADOC Reg, 351-4). Fort Monroe, VAt< 
Headquarters, United States Army Training and Doc- 
trine Command, February 1979. 

This .*o -ument defines Army policy, objectives, procedures, and require- 
ments for the conduct of job and task analysis* TRADOC Pamphlet 350-30 

is tho basis for the regulation, 

.* * 

V tj ji Training and Doctrine Command* Job and task analysis 

handbook (TRADOC Pamphlet 351-4)* Fort Monroe, VA; 
Headquarters, United States Army Training and Doc- 
trine Command, February 1979, 

The handbook details procedures for job analysis, task analysis, and 

the selection of tasks for twining* The handbook was designed as a 

reference for analysts and not as a complete authoring aid to* prepare 
trainees to perform job and task analysis procedures. 

P 1.1*12 Washington State Department of Personnel. Task analysis 
handbook * Olympia, ttfti Washington State Department 
of Personnel, July 1973* (PB 226 682) 

This source presents a detailed task analysis procedure for personnel 
analysts which includes examples of the forms task analysts use* Un- 
fortunately it falls short of an authoring aid because it does not step 
a task analysis trainee through the procedures* 

T 1.1<13 v aney , J . P. A critical review of the instructional 
technology mechanism of task analysis. Improving 
Human Performance: A Research Quarterly , . 1973, 3(2), 
64-70. 

Yaney offers a critical review of the instructional technology mechan- 
ism of task analysis. He discusses asstrts and liabilities of three task 
analysis techniques: ( 1 V cognitive analysis, (2) master performance and 
{ 3} field analysis* Yaney presents an operational cost model for task 
analysis. He concludes that the costs typically outweigh the benefits* 
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Status 

A variety of techniques and procedures exist for task and job analysis 
(Control Data Corporation, 1977 j Ammerman, 1977* Folley, ^964? and 
Powers, 1977)* * The notion of devising an inventory of job tasks is 
particularly well developed* A clearinghouse for the dissemination of * 
tas£ inventories has been established (TIE) as well as a compu^t^based 
system (CODAP) to analyze data gathered by job inventories* Tile TRADOC 
Pamphlet (35 1-4) indicates the job and task analysis data which must be 
collected but does not provide the analyst with the rationale for making 
the decisions involved in gathering this data* An authoring aid for this 
block is unavailable* 
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Purpose of Block I«2 Select Tasks/Functions 

To determine the tasks to be trained based upon certain selection 

criteria and resource availability* 

References / , 

P 1.2.1 Ammerman, Identifying relevant job performance 

(Vol* 3>* Performance 'Content for Job Training * 
Columbus j The' Center for Vocational Education, The 
Ohio State University, 1977. (ED 146 371) 

* 

Ammerman, H* L* & Essex, P. W* Deriving performance re- 
quirements for training (Vol* 4). Performance content 
for job training * Columbus: The Center for Vocational 
Education, The Ohio State University, 1977, (ED 146 372) 

Volume 3 provides a procedure for conducting surveys to determine 
relevant job tasks* The details offered on the types of survey ques- 
tions to be devised are useful. Volume 4 presents a task selection 
procedure employing prediction equations which determine the frequency 
of task performance* 

T 1*2*2 Page, E. B* , Jarjoura, D* Konopka, C* D* Curriculum 

design through operatlAis research* American Educa- 
tional Research J our pal , 1 3 ( 1 ) , 1976, 31-49* 

The authors present a technique which calculates values for the compo- 
nents involved in instructional decision* making* This would make a most 
useful authoring aid considering the required activities of Block 1*1* 
According to Page et al« operations research has rarely been used to 
determine what to teach* 

A 1*2.3 Schulz, R. E. Job aid for select tasks for training 
(UUMRRO Draft Manual)* Alexandria, VA: Human 
Resources Research Organisation, September 1978* 

Schul2 provides an authoring aid for Block 1*2 following the ISD text* 
A system of worksheets for the collection of data, which supports selec- 
tion decisions, appears to be a^ good technique for organising a great 
deal of task analysis input* 

Status 

Page et al* (1976) offer a technique for selecting tasks for training* 
Further development could make this a useful authoring aid* Ammerman 
and Essex provide procedures for task selection* An authoring aid fol- 
lowing the ISO text is offered by Schulz (,1978) 
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Purpose of Block X.3 n Construct Job Performance Measures 
To devise job performance measures (JPMs) for each task selected for 
training* JPMs are tests which indicate how well an individual can 
perform a job* * 

References 

T 1*3*1 Ammerman, H. L. & Essex* D. Deriving performance re- 

quirements fo* training {Vol* 4}* Performance content 
for job training . Columbus: The Center for Vocational 

f Education, The Ohio State University, \971. (ED 146 372} 

Volume 4 provides a brief introduction to an algorithmic method for 
calculating level of task performance* 

P 1.3*2 Gael, S. Employment test validation studies. JSAS Cata- 
log of Selected Documents in Psychology , 1974, 4, 95. 
(Ms- No. 7 11) * 

Gael discusses a procedure for the design and selection of instruments 
used to test clerical performance of telephone company employees. 

A 1.3.3 SchuU, R._E. Job aid .for conducting task analysis (HumRRO , 
Draft Manual). Alexandria, VA: Human Resources Re- 
search Organization, September 1978. 

Although the title differs from that of £lock 1.3 the activities de- 
scribed by this authoring aid are those of *1.3. The aid provides a hand- 
hook for collecting and recording information which operationalizes each 
job task. rt * 

Status 

A procedure exists for the developmenc of job performance measures for 
clerical staff (Gael, 1974)/ It is reasonable to expect that this pro- 
cedure could be applicable in .other vocational settings* An authoring 
aid exists following the procedures outlined in the ISD model (Schulz, 
'973) . _ , 
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P^jpose of * Block 1*4 Analyze Existing Courses 

To determine the decree to which a course includes the same tasks, which 

the command needs to teach* 

References 

A U4J Schulz, R. E* Job aid for analysing existing courses 
{ HumRRO Draft Manual). Alexandria, Vkt Human 
Resources Research Organisation, September 1978. 

V 

kchuls provides an authoring aid for Block 1.4 of the ISD model. 
Status ^ 

An authoring aid exists for this block which was designed following the 
guidelines of the ISD model {S£luilz # 1978). Procedures and techniques 
other than those specified in the* ISD text itself were unavailable. A 
validation of the Schulz aid would be particularly valuable for determining 
further developmental efforts for this block. 
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Purpose of Block 1*5 Select Instructional Settings 

To assign tasks to rme of the following instructional settings;' {1) 
Job Performance Ax is, (2) Self Teaching Exportable Packages, (3) Formal- 
on-the Job Training, ( 41 Installation Support Schools or (5) Resident 
Schools* 

* 

* 

A t .S . 1 Srhu Iz, R. E * Job aid for selecting instructional settings 
( HumKHQ Draft Manual ). Alexandria, VA: .Human Re- 
sources Research Organization, September 1^78. 

Schulz. provides an authoring aid which guides the trainee through Block 
of the ISD model* f - " * , 

Status 

Tho purpose of this block is unujut* to military instructional settings 
which limits the availability of references. Schulz ( 19?8. has developed 
an author m»j aid covering the five settings stated in the purpose of this 
block. Techniques indicating the distinctions and characteristics * of the 
f iw sot»Mn«is identified by the model and procedures for assigning tasks 
to all of these settings would be useful* Authoring aids following other 
instructional wo ie is would be of interest* * 
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PurpQ.se of Block 11*1 Develop Objectives 

To develop learning objectives for the learning categoriee of informa* 
tion mental akille* physical skills, and attitudes. 

References 

P 2*1*1 Gibbons, 'A. 5. & Hughes, J. A* A method of deriving hier- 
archies of instructional objectives. Proceedings of 
the Human Factor a Society Annual Meeting * Detroit # 
1978. 

Gibbons and Hughes offer^a procedure for transforming a taak list into 
hierarchies of instruct IotsI objectives* This document is a detailed 
authoring procedure which utilizes terminology of the ISD model* It 
would be a good candidate for an authoring aid effort* 

P 2*1.2 Hillelsohn, M* J* Inatructional objectives* Teaching 

with computers t A guide to creating instruction for 
computer presentation * Unpublished manuscript, 1979. 
(Available from Human Resources Research Organization, 
Alexandria, VA* ) 

Hillelsohn provides a procedure for devising objectives for computer- 
based learning* fcith the inclusion of opportunities for practice this 
document could be readily modified into an authoring aid. 

T 2*1*3 Roudabuah, G* E. An empirical structure for reading ob- 
jectives* A paper presented at the annual meeting 
of the American Educational Research Association * 

Chicago, April 1974. (ED 09S 194) 

Roudabush 1 * research examines hierarchical organizations of reading ob- 
jectives. Roudabush £ri#fly discusses instructional objective litera- 
ture and provides considerable detail about reading behaviors* 

A 2*1.4 Schulz, R. E* Job aid for developing objectives (HumRRO 
Draft Manual)* Alexandria, VA* Humar Resources 
Research Organisation, December 1978* 

This document is an authoring aid which follows the procedures outlined 
in Block XI.I-Develop Objectives* 
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Status 

\ * 

The Gibbons and Hughes (1978! effort' and the Hillelsohn (1979) effort 
/are useful procedures for developing objectives* The latter is particu- 
I larly of interest because it is a procedure for devising objectives from 
a task list which is the focus of this block* Both could be modified to 
form authoring aids* The Schulz (1978) aid follows the ISO model. 




Purpose of Block II. 2 Develop Tests 

To write test items for each learning objective* 

References 

P 2*2*1 Conoley, J., & O'Neil, H. F* , Jr* A primer for develop- 
ing test items* In H* F. O'Neil, Jr. (EdOi Procedures 
for instructional systems development * New York: 
Academic Press , 1979* 

Conoley and O'Neil provide detailed procedures for the construction of 
multiple-choice test items* Guidelines for revisions are embedded in 
test item examples* 

P 2.2*2 Gronlund, N* Preparing criterion^referenced tests for 
classroom instruction * New Yorki Macmillan, 1973* 

Procedures for the development of criterion-referenced tests are pro- 
vided* Groniund discusses the decisions which must be made by the test 
developers— i.e . , the level of mastery desired, scope of the subject mat- 
ter covered, and type of test item* Also indicated are practical rules 

for writing test items, 
* 

P 2.2*3 Plumlee, L. B. A short guide to the development of per- 
formance tests (Professional Series Pamphlet 75-1)* 
Washington, DC: t T *SW Civil Service Commission, 
Test Services section of the Personnel Research and 
Development Center, January 1975, 

Plumlee provides an overview of a procedure for development of psycho* 
motor tests. Task analysis, test reliability, and validity are also dis- 
cussed. * This source directs tftfc reader to more detailed refers :es. 



A 2.2.4 Roid, G. & Haladyna, T. A handbook on item writing for 
criterion-Sreferenced tests * San Diego, Chtf Navy 
Personnel Research and Development Center/ 1979* 

Roid and Haladyna provide an authoring aid for the development of criterion- 
referenced tests* The authoring aid has been designed to be compatible 
with the ISD model* 

A 2*2. S Schulz, R* E*, Hibbits, N* , Wagner, H* & Seidel, R* J, 
On-line authoring aids for instructional design 
(Technical paper) • Alexandria, VAi U»S. Army Re- 
0 search Institute for" the Behavioral and Social 
Sciences, in press* 

Schulz et al* provide an authoring aid for the development of test i^6ms 
following Block II. 2 guidelines* 
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A 2*2** Swesey, R* W* & Pearlstein, R* B* Guidebook for develop- 
ing criterion-referenced tests (Report No* 287-aR18(2)< 
IR-0974K Arlington, VA: Army Research Insti- 

tute for the Behavioral and Social Sciences, August 
1^75* CAD A014 987) 



Th** atanual ievised by Swezey and Pearlstein is a well developed author*- 
xnq aid for t^st construction in an Army environment. 

Status 

T*st item teveiopment is replete with procedures and authoring aids from 
civilian ani military environments* The Roid and Haladyna <1978) hand- 
hook and the Schuiz authoring aid have been designed with attention to 

the IS! motel. 
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Purposes of Block II V 3 Describe Entry Behavior 
1. To develop and administer a test of entry a 



tion according to the results* 



test of entry skills and revise instruc- 



*\>2. To devise a pretest which measures the trainee^ current knowledge 
>b> that instructional units mastered may be bypassed* 




References * 

No references could be identified for Block IX»3* 
Status 

The test development aspects of this block are most likely covered by 
procedures and authoring aids identified in Block I 1.2* Procedures need 
to be defined to identify the trainee 1 s entry skills* Authoring aids can 
then be developed* 
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Purpose of Blook 11% 4 Determine Sequence and Structure 

To devil* a learning hierarchy which specifies the sequence in which 

instructional objectives must be leernad* 



References 

P 2*4*1 Airasian,* P» W* k method for validating sequential in- 
- structlonal hierarchies* Educational Technology * 
December 1971, 54-56 • 

Airasian provides an explanation. of the notion of an instructional hier- 
archy • He presents a model for testing the validity of ah established 
hierwflhy of instructional tasks* This procedure is neadedj however, . 
a procedure for determining the interrelationships between tasks is a 
higher priority which is not addressed* 

T 2*4*2 Airasian, P. W. * Bart, w» M* The use of ordering theory 
in the identification and specification of instruct 
% tional sequences* A paper presented at the annual 

meeting of the American Educational Research Associa- 
tion . Chicago, April 1974* 

■J 

Airasian and Bart review a technique, which is an ouigjrowth of Guttman, 
for ordering instructional tasks* As it. stands this document is too 
complex* A simplified version of ordering theory would be useful for 
an authoring . aid* 

T 2*4*3 Boozer, R* F* & Lindvall, C* M* An investigation of 
^ selected procedures for the development and evalua- 

tion of hierarchical curriculum structures* A paper 
presented at the annual meeting of the American Edu- 
cational Research Association * February 1971* 

A study assessing the usefulness of Guttman f s sea logr am analysis and sim- 
plex analysis for a' mathematics program is presented* these techniques 
provide procedures for investigating the sequence of instruction in terms 
of student performance* Boozer and Lindvall af f i rm the value of sea logr am 
and simplex analysis, however, this report lacks sufficient detail for 
the novice instructional developer* 

P 2.4.4 Dansereau, 0. R. , Evans, S. H., Wright, A. D., Long, G. 

& Actkinson, T. Factor* relate d to developing in- 
structional Information sequences* Phase I (Report 
No. AFHRL-TR-73-5KI)). Lowry Air Forca Baaa, 
CO: Air Fore* Human Resources Laboratory, Technical 
Training Division, March 1974. (AD 777 832-) 

oans«r«au, D. R«, Ivans, S. H. , Actkinson, T. A*, and 
Long, G. L. Factors relating to the development of 
optimal instructional sequence a (Report No. AFHRL- 
TR-73-5KII)). Lowry Air Forca Basa, CO: Air 
Fore* Human Rasourcas Laboratory, Technical Train- — ' 
ing Division, June 1974. (AD 783 843). 
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Phase I provides a review of instructional sequencing literature* The 
results of an experiment conducted in Phase II indicate that a multi-* 
dimensional scaling procedure for generating instructional sequences was 
useful and reliable. 

T 2.4.5 Dansereau, D* F* , Long* G* L*, Evans, S* H*, & Actkinson, 

T* R* Objective ordering of instructional material 
> * using multidimensional scaling* Journal of Structural 

Learning *, in press* 

Dansereau ,et al* discuss multidimensional scaling as a technique for de- 
vising instructional sequences -and the interactions of sequencing with 
type of material and individual differences* A good review of sequenc- 
ing literature is provided but iwis insufficient for a procedure or an 
authoring aid* 

P 2.4.b Durell, A* B* A computer simulation study of measures 
for validating learning hierarchies. A paper pre* 
sented at the annual meeting of the American Educa- 
tional Research Association , Chicago, April 1974* 

Dur*ll examines procedures for validating a hierarchy of^tasks* 

P 2.4*^ Hilielsohn, M* J* Organizing instruction. • Teaching with 
computers: A guide to creating instruction for com-* 
puter presentation * Onpublfshed manuscript, 1979* 
(Available from Human Res6urces Research Organisa- 
tion, Alexandria, VA*) 

A procedure for structuring and sequencing* computer-based learning mate* 
rials is provided* 

T 2.4*9 McDade, C* E* Subsumption versus educational set: Impli- 
cations for sequencing of instructional materials* 
Journal of Educational Psyche y, 1978, 70(2), 137- 
141*^>ED 187 983) 

MdDade tested Wo sequences of educational psychology instruction* A good 
review of the literature on instructional sequencing is provided* 

P 2*4*9 Phillips, E* 9* Development of optimal Instructional se- 
quences* A paper presented at the annual meeting of , 
the American Educational Research Association , Chicago, 
April 1974* (ED 097 374) 

Phillips reviews research supporting the notion that there can be an opti- 
mal sequence of instruction* n roc*»dures for developing and validing in- 
structional sequences are also presented* This is a readable literature 
review which could supplement the ISO text* 
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T 2.4.10 Reigeiuth, C. M» In search of a better way toorganiEe 
instruction* The elaboration theory* Journal .of 
Instructional Development , V979, £<3), 8-15, 

Reigeiuth provides a brief introduction to the elaboration model of 
sequencing instruction* The model suggests that instruction should be * 
organized in a M general-to-detailed° pattern rather than a hierarchical 
sequence, A user's manual is in press, \ 4 

T 2*4*11 Seidl*. N. W. & McKeen, R. L. More on the use of student 

generated learning hierarchies* Improving Human Per- p 

formancet A Research Quarterly , 1974 3(2), 71-89* 
- ^ _ 

Seidl and Mefceen examine learning hierarchies for algebra instruction 
devised by low and 'high ability students. Although significant results 
were not obtained, the notion of students devising their own learning 
hierarchies deserves more attention in light of current efforts to indi- 
vidualize instruction. 



T 2*1, 12 wellens, B. , Lenke, J* ft. & Oswald, J, H. An investiga- 
tion of a proposed hierarchy of mathematics concepts 
and skills using ordering theory* A paper presented 
at the annual meeting of the American Educational 
Research Association * New York City, March 1977* * 

The ordering theory approach to the construction of instructional hier- 
archies is reviewed. Wellens et al. investigated the effects of a 
hierarchy of mathematical skills and concepts generated by the order- 
ing theory technique. 



Status ; 

* 

Procedures for devising learning hierarchies exist (Dansereau et al. , 
1974, in press; Dureli, 1974; Hiilelsohn, 1979). Research studies have 
examined optimal sequencing of instruction (Phillips, 1974) and the devel- 
opment of learning hierarchies by students (F*idl and McKeen, 1974). A 
variety of theories exist for sequencing instruction; however, no reference 
approaches an authoring aid* 
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Purpoit of Block 111,1 Specify Learning Events/Activities 

To classify learning objectives and identify "learning guidelines 

necessary for optimum learning to taka place* 91 



References 

T 3,1,1 



Aagard* J* A* & Braby, R, learning guidel inaa and 
algorithms for types of training objectives (TAEG 
Raport No, 23), Orlahdo, FX. t Training and Analy- 
sis Group, March 1976, Nad A023 066) 



Aagard and Braby hava identified 11 classes of instructional objectives 
•(such aa rule learning) and have devised a training atrategy for each 
class. The strategies specify learning events which are compatible 
with the instructional objectives, 

A 3,1,2 Courseware, Inc, Author training course , San Diego, 

CA: Courseware, Inc, January 1978, Also In H, F, 
O f Neil, Jr, (Ed,), Procedures for instructional 
systems development * New York: Academic Press, 
1979, 

The Author Training Course is a series of authoring aids designed to 
assist course authors in converting instructional objectives into in- 
structlonal materials* The course follows the guidelines of the 1SD 
model* Particular attention is paid to developing instruction which 
maximi zes learning. 

Status 

The Aagard and Braby (1976) technique would make a useful authoring aid. 
The second reference available for this block is a course designed for 
course authors who are tasked with Block 111,1 of the 1SD mod^l* 
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Purpose of Block II I. 2 Specify Instructional Management Pitt and 
Delivery 8yetam 

1* To apecify the delivery ayatam through media aalaction. 

2. To prepare organisational plan for the inatructlon which lndicataa 
inatructional actlvitiee of atudanta and inatructora. 

Reference* 

P 3.2.1 Braby, R. , Henry, J. M. , Parriah, W. F. , * Swope, W. M. 

A technlqua for chooainq coat-affactlva inatructional 
' da 11 vary ayatama (TAEG Report No. 16). Orlando, FL: 
Training and Analyaia Group, April 1975, Reviesd 
October 1978. 

The Training Ef fectiveneaa, Coat Ef fectiveneaa prediction Technique 
(TECEP) la a procedure deeigned to aaaiat experienced training apecial- 
ieta in the aelection of Inatructional delivery ayatens. The aelection 
la baaed upon the madia requirementa of the training objactivea and the 
coat-ef fectiveneaa of the varioua delivery ayatama which meet th«ae 
requirementa { aee T 3.1.1). 

P 3.2.2 Bret*, R. The aelection of appropriate communication 

media for- inatructlon: A guide for dealgnera of Air 
Force technical training programs (Report N©» R-6C1-PR). 
Santa Monica, CAi RAND Corporation, February 1971. 
(AO 721 443) 

Bretz diacuaaea a procedure for media aelaction which ia baaed on learning 
objactivea, course content, student character istica, and teaching method. 

T 3.2.3 Department of Defsnee Audioviaual Information Syatam. A 
user guide for the audiovisual product! data base. 
Arlington, VA: Office of Information for the Armed 
Forcee, DoD Directorate for Audiovisual Activities, 

*,975. 

This document is a user guide for a computer-based network of audio-visual 
producta. The guide ia largely concerned with the manipulation of the 
Haieltine terminal which ia the delivsry mechaniem for th« system. 

T 3.2.4 Keeler, F. L. * Rlsso, W» A. An evaluation of microfiche 
reader types for uaa with programmed instruction (TAEG 
Report No. 35). Orlando, FLi Training Analysis and 
Evaluation Group, August 1976. (AD A029 714) 
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One study compared several types of microfiche readers and indexing 
techniques, in an attempt to explore ease of reader usage with Navy re- 
cruits * A second study examined differences in efficiency with the use 

of microfiche versus traditional paper copy* No significant differences 
in performance were observed* The studies answered important questions 
concerning microfiche media but did not approach an authoring procedure 
or aid* 

P 3*2*5 Lindahl, w* H* & Lin, B. An heuristic approach for 

the scheduling of Navy Training Courses ( TAEG Report 
No* 52). Orlando, FL: Training Analysis and Evalua- 
tion Group, December 1977* (AD A048 183) 

A procedure indicating steps and sources for scheduling Navy training 
sources is presented* The advantages of automated scheduling are 
examined. 

T 3.2.6 Lonigro, J* F» & Eschenbrenner , A* J., Jr* A model for 

selecting the media in technical/vocational educa^pn* 
In H* Hitchens (Ed*), Selecting media for learning: 
Readings from audiovisual instruction * Washington, 
D*C*: Association for Educational Communications and 
Technology, 1974. 1 

Lonigro and Eschenbrenner propose a model for media selection which con- 
siders types of learning required by task, desired level of mastery, 
matches between types of learning and media, and media production costs* 

T 3*2.7 Ruefli, T. W* # Ruefli, J* C*, 0*Neil, H*, Jr*, £ Smith, 

CM* A scheduling strategy for computer-managed instruc- 
tional systems* Socioeconomic Planning Science , 1978, 
22, 145*152* 

Ruefli et al* describe a technique for and a test of a scheduling sys- 
tem for computer-managed Air Force training* The scheduling system 
assigns the student to the next lesson based on previous performance 
and computer terminal availability* ^ 

A 3*2*8 Schulz, R* E* Job aid for specifying instructional man* 

augment plan and delivery system CHumRRO Draft Manual)* 
Alexandria, VA: Huptan Resources Research Organization, 
January 1979. 

Schulz has devised an authoring aid which guides an author through the 
procedures specified by Block 111*2 of the ISO model* 

P 3.2*9 Smith, E. H* Quality assurance of media devices and 

courseware ( AFHRL-TR-75-37) * Lowry Air Force Base, 
CO: Technical Training Division, Air Forc# Human 
Resources Laboratory, October 197S* CAD A018 B67) 
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Smith presents procedures which assure quality in the development of 
visuals and recordings* Procedures to select cassettes and films are 
briefly discussed. Smith focuses on specifics of good cassette and 
film production* 

T 3. 2r-K) Spangenberg, R* W. Overview of mediated courseware for 
- learning centers (AFHRL-TR- 76-37 ). Lowry Air Force 
Base, CO: Technical Training Division/ Air Force 
Human Resources Laboratory, June 1976* (AD A033 304) 

Spangenberg presents guidelines for media selection based on the type 
of information which is to be learned (i*e*, intellectual skill, cogni- 
tive strategy, verbal information, motor skill, attitude)* An effort 
to rewrite this document following ISD terminology would make it a 
useful procedure or authoring aid* 

A 3.2. It Stolovitch, H. D* Systematically selecting media: A 

mathematical procedure* NSPI Journal , 1977, 16 ( 7 ) , 
11-15* 

Stblovitch provides an authoring aid for devising "yo^ir own" media/media 
attribute matrix for media selection. A mathematical analysis of a com- 
pleted matrix provides input for decision-making which incorporates (1) 
the range of media available and (2) the media characteristics required 
for attaining training goals. 

Status * 

Procedures and techniques exist for the selection of media based on learner 
characteristics (Bretz, 1971) and type of learning required by the task 
(Lonigro & Eschenbrenner, 1974; Spangenberg, 1976; Bxaby at al. # 1978). 
Several documents examine the use of different types of audiovisual media 
in depth including cassettes, films* and microfiche* An authoring aid 
exists for media selection which considers f type of media and desired 
learning outcome (Stolovitch, 1977)* Another intention of Block III. 2 
is to devise a schedule for the progress of instruction* A procedure 
and a technique exist which provide guidance on scheduling traditional 
and computer-based courses (Lindahl & Lin, 1971? Ruefli et al. , 1978). 
The Schulz (1979) authoring aid covers media selection and the prepara- 
tion of an organizational plan* 
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Purpose of Block III* 3 Seview/Select Existing Materials 
To review existing instructional materials and examine the corres- 
pondence with instructional objectives of the current training need* 

References 

P 3.3.1 Merrill, M. D., Reigeluth, C. M. ii Faust, G* W. The 

instructional quality profile. In H. F. Q'Neil, Jr. 
( E * Procedures for instructional system develop- 
ment . New York: Academic -Press, 1979. 

Mttmli *»t al. discuss a procedure for diagnosing and correcting weak- 
nesses in existing instruction through the examination of instructional 
obiectivos. An authoring aid for this procedure is in production 
<P 3.4.11, P 5.3.1). 



A 3.3,2 SchuU, R\ E. 



Job aid for review and selecti on of 

Alexandria, 



oxisting material (HumRRO Draft Manual )• 

VA: Human Resources Research Organization, October 
1^?H.' 



Schul?. has prepared an authoring aid for Block .111.3 of the ISD model 
which follows the procedures outlined in the ISD text. 



Status 

* 

Th*> .iuthorin«? ai l for this block, which is now in production (P 5.3.1) 
will provide guidance in the analysis of existing instruction. The 
authoring aid provided by Schulz (1978) follows the procedures of the 
ISD mo i«* U 
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Purpose of Block III. 4 Develop Instruction • 
To prepare new instructional materials for learning objectives which 
are not covered by existing materials* 

A 3 • 4 • 1 Sell* N. T* & Abedor, A* J. Developing audio-visual in~ 
structional modules for vocational and technical 
training . Eftglewood Cliffs, NJ* Educational Tech- 
nology Publications , 1977* 

Bell and Abedor provide an authoring aid for the development of self- 
instructional audio-visual materials based on specified instructional 

objectives* 

A 1*4.2 Braby, R. , Parrish, W. F. , Guitard, C* R* & Aagard, J. A* 
Computer-aided authoring of programmed instruction 
for teaching { TP EG Report No* 58). Orlando, FL: 
Training Analysis and Evaluation Group, June 1978. 

\ 

This report describes a computer program (AUTHOR) that generates pro- 
grammed training manuals which teach symbol recognition* The present 
document details computerised authoring system which produces a learn* 
package containing a pre-test, drills, and criterion tests based on 
author input of a set of symbols. The next two sources are companion 
documents to a 3.4.2 and therefore do not have reference numbers. 

nui tard, C. R. Computer-aided authoring, system (AUTHOR) 
users guide, volume I (TAEG Report No* 59). Orlando, 
FL: Training Analysis and Evaluation Group* June ( 
1978. * * 

This document is an operator* s manual which accompanies the previous 
reference. "It also lists the computer program for AUTHOR which assists 
in th* preparation of training materials* 

Braby, R. Computer-aided authoring of instructional 

materials (Tech* Note 2-79). Orlando, FL: Training 
Analysis and Evaluation Group, June 1978. 

Braby rev. ws the computer-aided authoring technique presented in TAEG 
Reports No* 58 and 59. He also reports that materials produced by coin* 
puter-aided authoring routines were found to produce performance superior 
to those using traditionally developed materials* 

A 3.4.3 Braby, R* , Kincald, J. P., & Aagard, J. A. Use of 
mnemonics in training materials; A guide for 
technical writers (TAEG Report No. 60). Orlando, 
Fhi Training Analysis and Evaluation Groups 
July 1978. * 
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Braby et al* provide an authoring aid for writing mnemonics (memory aids 
to be included in training materials for classroom instruction* Nine 
mnemonic techniques are reviewed and details on when each technique - 
should be used are given* The following document is a companion docu- 
ment , to A 3*4*3. 

Ainsworth, J* S. Symbol learning in Navy technical 
training (TAEG Report No. 66). Orlando, FL: 
Training Analysis and Evaluation Group, January 
1979, 

This report is an s evaluation of the technique developed by TAEG for 
writing mnemonics (Report No. 60) for symbol learning and a feasibility 
demonstration of the computer-based authoring system (a 3.4.2). Hand- 
books produced by the computer-aided authoring routines were found to 
be iesi rahie as well as the complete algorithm for symbol learning 
during mors* code training* 

A 3.4.4 Courseware, Inc. Author training course (Data Item No. 

0002AD). San Diego, CA; Courseware, Inc., January 
1978. ^Alao in H. F. O'Neal, Jr. (Ed.), Procedures 
for instructional systems development . New York: 

^ Academic Press, 1979. 

The Author Training Course is a aeries of authoring aids designed to 
guile course authors through Block ill. 4 of the ISD model {A 3.1.2). 
Th*> goal of the course is to have authqrs develop instruction for 
specified instructional objectives* 

A 3.4. S Deter line, W. A. £ Lenn, P. D. Coordinated instructional 
systems . Palo Alto, CA: Sound Education, 1972* 

This is a, lesson book which forms a self study course on instructional 
design. The book is a thorough authoring aid which guides the author 
through the development of two lessons., Dete^line and Lenn indicate 
the time needed to reach mastery in their course of instructional 
design ts an average of 35 hours. 

1 

P 4 3.4*f> Hiilelsohn, M. J* Production of CBLM. Teaching with 

computers* A guide to creating instruction for com«» 
puter presentation * Unpublished manuscript, 1979. 
(Available from Human Resources Research Organization* 
Alexandria # VA*. ) 

Hiilelsohn presents a procedure for putting instructional materials 
into a computer system* Th€* steps involved in programming, inputting, 
and debugging the system are detailed* 

P 3.4*7 Joyce, R. P* # Che2noff, A. P* ,* Mulligan, J. F. & Mallory, 
W* J. Fully procedural! zed job performance aids? 
Handbook for 3PA developers (Tech. Report)* Vallencia, 
PA: Applied Science Associates, December 1973* 
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Joyce, R. P . , Cheznc^ff, A* P*, Mulligan, J. F. * Mallory, 
W. J • Fully prdceduralized job performance aids: 
Handbook for JPAVgianagers and training specialists 
(Tech* Report)* VaTTencia, PA: Applied Science 
Associates, December 1973 • (AD 775 706) 

Joyce et al. provide detailed procedures for the development of Job Guides 
and Troubleshooting Aids for Air Force equipment maintenance, Ttie devel- 
opment of this information into an authoring aid would give input for 
those tasked with Job Performance Aid development. 

A 3.4.0 Kern, R. P. , Sticht, T. G., Welty, ,D. & Hauke, R. N. 

Guidebook for the development of Army training liter- 
ature . Alexandria, VA: Human Resources Research 
Organization, November 1976. (AD A033 935) 

■* 

This document is an authoring aid f6r the development of training 
manuals* The guide presents examples of training manual problems in a 
"before and after revision" format. 

A 3.4.9 Schulz, R. E. , Hibbits, N. , Wagner, *H.v £ Seidel, R* j # 

On-line authoring aids for insCructtional design (Tech- 
nical paper). Alexandria, VA: U.S. Army Research 
Institute for the Behavioral and social Sciences, in 
press. - \/ \ 

An authoring a^d is provided for Block III. 4 of the ISD manual. 

A 3.4.10 Shriver, E. l» Fully t j>roce3uraIi2e4 jjcrift performance aids: 
guidance for performing behavioral analyses of tasks 
( AFHRL-TR-?5-38). Lackland Air Fofice Base, TX: Air 
Force Human Resources Laboratory, Occupational Research 
Division, June 1975. (AD A015 059) 

y 

Shriver ameada , the Joyce et al. (see P 3.4.7) documents with finer details 
for job performance aid development. 

P 3.4.11 Wulfeck, W. H. , Ellis, J. A. , Richards, R. E., Wood, 
N. E. fi. Merrill, M. D. The instructional quality 
inventory: I. Introduction and overview (Special 
Report No. 79-3). San Diego, CA: Navy Personnel 
■ Research and Development Center, November 1978. 

The Instructional Quality Index (IQI) is a procedure for the development 
of*, instruction based on the learning behaviors and outcomes specified by 
the instructional objectives. Authoring aids for the IQI are in prepara- 
tion. The IQI was originally called the Instructional Strategy Diagnostic 
Profile (ISDP) (P 3.3.1, P 5.3.1). 
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Status 



This block is the best developed in terms of the availability of author- 
ing aids for print and audio-visual using the ISO model (Courseware, 1978? 
SchuU et al., 1977, Wulfeck et al. , 1978; Braby et al. 1978) perhaps 
because it is the most basic t,o the intent of the model. Authoring aids 
are available for the programming- (Hillelsohn, in press) and are unavail- 
able vor the development of computer-based instructional materials. 
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Purpose of Block 111,5 Validate Instruction 

To debug instructional materials b^ trial with students* 



R eferences 

P 3*5.1 Dick, W. Applications of formative evaluation to the 

instructional design process* A paper presented at 
the annual meeting of the American Educational Re- 
search Association . New York City, April 1977. 
(ED 132 135) 

Dick reviews procedures for formative evaluation which must be part of 
an instructional development plan so that evaluative information is 
gathered during each phase of the developmental process. 

P 3.5.2 Kan iaswamy, S., Stolovitch, H. D. & Thiagarajan, S. 

Learner verification and revision; An experimental 
comparison of two methods. AV Communication Review , 
1976, 2_4(3) , 316-328. 

Kan iaswamy et al. examined the cost effectiveness of two procedures for 
testing instructional materials on students and making revisions based 
on their feedback. Instructional materials can be piloted with one stu- 
dent or with a group^of students* Their research findings indicated no 
significant difference. These results irjdicate that the more cost ef- 
fective method would be trials with one student at a time* 

'TV 3 . ^ . ^ Rayner, G. T. An empirical study of a methodology for 
the revision of systematically designed educational 
materials (CAl Center Tech Report No* 24)* Talla- 
hassee, FL: Florida State University, March 1972. 
(ED 067 87?) 

Rayner developed and tested a model which identified revisions for in- 
structional materials based on content and procedural deficiencies* 
Th*» model was found to be empirically useful for revising a health edu- 
cation course. (See Block V.3.) 

A 3.5.4 Schulz. fc* E* Job aid for validating instruction { HumRRO 
Draft Manual). Alexandria, Vht Human Resources Re- 
♦ search Organization, October 1978* 

Schulz provides a detailed authoring aid* which steps the trainee through 
validation procedures identified by the ISO model* 

p i.S.S Wulfeck, W* H,, Ellis, J. A*, Richards, R* £* , Wood, N. E. , 
& Merrill, M* D* The instructional quality inventory; 
I» Introduction and overview (Special Report No. 79-3). 
San Diego, CA; Navy Personnel Research and Development 
Center, November 1978* 




The IQl is a quality control procedure designed to evaluate instructional 
objectives, tests and materials designed by the ISD method (P 3.3*1, 
P 5.3. 1). 



Status 

» * » 

Procedures, techniques, and authoring aids following the ISD model (Schulz, 
1978? Wulfeck et al. , 1978) exist for pilot; testing newly developed instruc- 
tional materials. 



PHASE III 
DEVELOP 



\ 



in j 

Specify 
learning A 
Events/ ^ N 
Activities' 



,111,2 

Cspeci fy 
^Instruction 
^Management 
>1an & Deli- 
very System 



3 



Mil. 3 
^Review/ 



Select 



^Existing 

x Materials\ 



A P 




III. 5 ~ 
Val idate 
Instruction 



APT 



U 

37 



I 



Purpose of Block IV, 1 Implement Instructional Management Plan 
To initiate plan which specifies trainees, learning activities, and re- 
sources (this is achieved by making and completing checklists). 

References 

The authors could identify no references for this block* 
Status ; 

V 

Some references included in Block 111*2 discuss implementation* An 
authoring aid would be useful for activities such as the development of 
checklist used during implementation* 
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Purpose of Block IV, 2 Conduct Instruction 

For the instructor to initiate the instruction using the setting, methods 
and media specified by the management plan* 

References 

P 4*2. 1 Hilielsohn, M* J. Implementation of CBLM* Teaching 

with computers: A guide to creating instruction for 
computer presentation , Unpublished manuscript, 1979, 
(Available from Human Resources Research Organization, 
Alexandria, VA. ) 

Hilielsohn provides a procedure for initiating computer-based instruc- 
tion by detailing the tasks of the course administrator. 

Status 

The Hilielsohn (1979) procedure would make a useful authoring aid for 
computer-based instruction. Procedures and authoring aids are needed 
which focus on the implementation of traditional classroom instruction 
utilizing a variety of media. 
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Purpose of Block V*1 Conduct Internal Evaluation 
1* To determine student progress* 

2. To improve quality of instruction* 

These goals are achieved through the collection of data from students, 
instructors, and supervisors and the evaluation of tiffs data* Data 
must indicate the relationship between student entry behavior and 
achievement on the terminal learning objectives* Itxe 2SD effort is eval- 
uated as well* The development of the instruction is monitored to see 

that it is consistent with prescx'ibed 1SD procedures* 

* 

References 

P 5.1.1 Borich, G* D. A systems approach to the evaluation of 
training* In H# F* O f Neil, Jt» (Ed*) , Procedures 
for instructional systems development * New York: 
Academic Press, 1979* 

Structured decomposition is a program evaluation procedure in which data 
is collected in each component of a program* Discrepancies observed 
between planned and actual outcomes are examined* 

P 5.1.2 champagne, A* B. & Klopfer, L* E* Formative avai^stion 

in science curriculum* Journal of Research in Science 
Teaching , 1974, VI (3), 185-203* (ED 088 718) 

A procedure for the formative evaluation of a new science Curriculum is 
presented. Sources of evaluation data considered are the curriculum de- 
velopment plan, student instructional materials, student behaviors with 
respect to science content, student behaviors in the classroom, the teacher 
and course marketability* 

P 5.U3 Elsbree,,A* R* & Howe, C* An evaluation of training in 
three acts* Training and Development Journal , 1977, 
(July), 1-19* 

Elsbree and Howe discuss procedures for the evaluation of training which 
focus on (1) the establishment of the general scope and objectives of 
the- evaluation, (2) the production of a blueprint for conducting the 
project and (3) obtaining data, interpreting data, and reporting to the 
client* Flow charts of each of these phases are provided as well as a 
narrative providing an .example specific to Civil Service training* 

T 5*1*4 Hall, E* R* , Lam, K., t Bellomy, S« Training effective- 
ness assessment; Volume I, Current military training 
evaluation programs (TAEG Report No* 39) * Orlando, 
FL: Training Analysis and Evaluation Group, 
December 1976* (AD A036 517) 
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Hall, E. R. , Rankin, w. c. & Aagard, J. A. Training ef- 
fect iveness assessments Volume II, Problem*, concepts, 
and evaluation alternatives (TAEG Report No. 39). 
Orlando, FL$ Training Analysis and Evaluation Group, 
December 1976. (AD A036 518) / 



Volume I examines interservice training evaluatiotf efforts. Volume II 
discusses problems which have been observed in Najvy training evaluation 
efforts. A variety of evaluation techniques are X«viewed with guide- 
lines indicating the most appropriate use of each.^-v 



P 5.1.5 



U.S. Civil Service Commission. Training evaluation; 
guide to its planning, development, and use in 



agency 



training courses (Pamphlet T-13). Washington, DC 
Bureau of Training, U.S. Civil Service Commission, 
May 1971. 

This pamphlet describes a procedure for assessing effectiveness of 
Internally developed and administered training courses. 



P 5.1.6 U.S. Civil Service Commission. A process for the evalua- 
tion of training . Washington, DC: Training Leader- 
ship Division, U.S. Civil Service Commission, April 
1978. 



This document provides a procedure for formulating and implementing an 
evaluation plan. 



Status 



Procedures and techniques exist for student assessment (Champagne 6 
Klopfer, 1974; Hall et al. , 1976). Procedures for planning internal 
evaluation are also available (Borich, 1979; Elsbree & Howe, 1977; U.S, 
Civil Service Commission, -1971, 1974). Unfortunately authoring aids 
and procedures which follow the ISD model are unavailable. 
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Purpose of Block V. 2 Conduct External Evaluation , * 

To determine whether trainees meet the needs of the command* 

The external evaluation examines survey data of trainees in the field, 
30b performance measures, and evaluation of supervisors* 

Reference s 

P 5 * 2 • 1 Department of the Air Force* Training and evaluation 

course reviews (ATC Regulation 52- 1)* Randolph Air 
Force Base, TXt Headquarters Air Training Command, 
Department of the Air Force, September 197 U 

This document is a regulation which details procedures for internal and 
external evaluation. The intention of the data collected is to provide 
feedback for ISD course improvement* 

P 5*2*2 Pennell, R. , Harris, D. , Schwille, J* Appraisal of Air 

Force training course field evaluation system ( AFHRL~ 
TR-76-63). Lowry Air Force Base, CO; Technical 
Training Division, Air Force Human Resources Labora- 
tory, October 1976. (AD A035 641) 

# 

Pennell et al. examined the Air Force field evaluation procedure with 
the intention of improving data analysis and reporting* Recommendations 
were made which focused on the design of the evaluation instrument and 
frequency of evaluations. 

P 5.2.3 Walker, R. A. Post-hoc evaluation of S~3A instructional 
systems development . San Diego: Courseware, Inc. , 
1978* 

Walker reports on procedures used to evaluate a training course for an 
anti-submarine warfare aircraft (S-3A) developed by an ISD approach. 

Status 

Procedures exist for conducting* an external evaluation for military 
training courses* Authoring aids are needed which follow the ISD model* 




Purpose of Block V.3 Revise System 

To revise the system based on internal and external evaluation reports 
and changes in instructional needs due to new doctrine, procedures , 
or weapons systems. 



References 

P 5*3.1 Ellis, J* A., Wulfeck, w. H. , Merrill, M. D. , Richards, 

R. E. , Schmidt, R. V, & Wood, N. D. Interim training 
manual for the instructional quality inventory (NPRDC 
TN 78-5)* San Diego, Chi Navy Personnel Research 
and Development Center, February 1978. 

The Instructional Quality Inventory is an analysis procedure for evaluating 
and revising instructional materials. This document, which is currently 
undergoing revision, is a manual designed to train instructional developers 
in the use of the IQI (P 3.3*1, P 3*4.11). 



T 5.3.2 Gropper, G. L. Diagnosis and revision in the development 
of instructional Materials . Englewood Cliffs, NJ: 
Educational Technology Publications, 1975. 



Gropper reviews types of student failure and pjf^gram failure which indi- 
cate needs for program revision. 



T 5.3.3 Rayner, G. T. An empirical study of a methodology for the 

revision of systematically designed educational materials 
(CAI Center Tech Report No* 24}. Tallahassee, FLx 
Florida State University, Marcy 1972. (ED 067 877) 

Rayner has designed a model for revising the content and procedures of 
instruction which incorporates instructional objectives, diagnostic tests, 
prescriptions, and posttests. The model has been successfully tested on 
health education courses. 



Status 



/ 



A procedure and technique exist for revising a bourse based on internal 
and external evaluation data. An authoring aid (Ellis et al. , 1978; 
P 3.4.11) following the ISD model is in preparation. More procedures and 
authoring aids would be valuable. 

r , 
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CONCLUSIONS 



The following section presents references for authoring systems* 
The intent of these systems is to provide authoring aids or procedures 
for all steps of the instructional systems design process • A summary 
is then provided indicating the availability of authoring aids, pro- 
cedures, and techniques by phase* Finally suggestions for research 
and development ar* made in light of the availability of such materials 
as noted in the summary* * 

Authoring Systems 

Several systems have been developed or are in development which 
guide an author through complete phases of an instructional development 
effort. The systems use a variety of media for delivery and generally 
incorporate the procedures identified by the 1SD model* 

Control Data Corporation* Computer resource for effective 
author training in education (CREATE) * Minneapolis, 
KN: Control Data Corporation, 1978. 

CREATE is a curriculum for the development of individualized instruc* 
tion. A variety of media is used to deliver the course including 
printed text, videotapes, and CAI. The student author is expected to 
take the course at a learning center so that all media are readily 
accessible. The curriculum is modularized so that a program of 
Instruction can be designed for each student. 

Department of the Air Force. Training handbook for designers 
of instructional systems (AFP 50-58). Washingtqn, IX: 
Department of the Air Force, July 1978* 

The Air Force training handbook is a set of authoring aids or proce- 
dures detailing an instructional systems development model ( AFM 50-2). 
The major steps of the model are: (1) Analyze Systems Requirements? 
(2) Define Education or Training Requirements! (3) Develop Objectives 
and Tests; (4) Plan, Develop, and Validate Instruction* and (5) Conduct 
and Evaluate Instruction. Instructional developer^ receive guidance 
in filling out a series of forms for each step of the ISO process. 

Hughes, J., O'Neal, A. F., Kearsley, G., O'Neal, H»# 

Fee, H« , Rodi, L» , 6 Hermanns, J# Functional specifi - 
cations to computer aided training systems development 
and management (CATSDM) support functions (Tech* Report 
N AVTRAEQU I PCEN 74-C-0018-1), Orlando, FL: Naval 
Training and Equipment Center, July 1978* 
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This reterence is a specification for a system of computer programs 
which manages the total ISO process* 



Harris, w# P. An authoring system for on-the-job ^environ- 
ments* In H # F. Ocelli Jr. <Ed«), Issues in instruc- 
tional systems development s 
1979* 



New York* Academic Press, 



The Lincoln Laboratory has devised and field tested a system which en- 
ables subject matter experts <SMEs) to create computer-deliverable on- 
the-job training materials. A Lincoln Terminal System wis used to 
deliver the authoring system* The authoring training lessons were 
created to guide SMEs through the development of CAI materials without' 
providing instructional development theories. 

* ■ 

Statu s 

A system incorporating all blocks of the ISO model is currently unavail- 
able* The Department of the Air Force Training Handbook (1978) is not 
directly applicable for Army training developers because of its reliance 
upon Air Force terminology. Tne Hughes et ai* (1978) authoring support 
system will be most appropriate to this need once it has been developed. 
Throughout the pri- t reference section a series of authoring aids pre- 
pared by Schulz (Hximan Resources Rfesearch Organization) have been noted. 
The aids have been developed under contract to the U.S. Army Research 
Institute for Behavioral and Social Sciences* By October of 1979 thir- 
teen aids covering Phases I, II, and III (excluding Block 1.1) will have 
been developed and evaluated. c 



Summary of Availability of Authoring Materials by ISP Phast* 
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Job information is collected and analyzed during Phase I. Tasks 

are identified for training and test* are developed to show job mastery. 
An analysis of existing courses is made to determine the availability 
of courses designed to teach the tasks selected for training. The best 
match between selected tasks and instructional setting is then determined. 

Schuls (A 1.2.3, A 1.3.3, A 1.4. 1 f A 1.5.1) has devised authoring 
aids for all , of the above activities except job analysis* The Schulz 
aids provide forms to be filled out covering each procedure identified 
by the ISO model* Although an authoring aid does not exist for Block 
1.1 -Analyze Job, there are procedures available which could be converted 
to an authoring aid format (Training and Doctrine Command, P 1.1.11). 
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The Design phase utilizes the job analysis data produced in Phase I. 
Learning objectives are devised for each task and test items are pro- 
duced for each learning objective. Pretests are developed to examine 
the trainees' knowledge r>rior to instruction so that the% level of in- 
struction is appropriate to the needs of its users. An analysis of t 
the learning objectives is made to determine the most appropriate order 
for the instructional activities. 

Authoring aids exist for the development of, objectives (Schul*, 
A 2.1.4), and test items (Roid & Haladyna, A 2.2.4; Schulz, A 2*2.5; Swezey 
Pearlstein, A 2.2.6). Procedures are available for sequencing instruction 
by creating learning hierarchies (Dansereau et al* * P 2.4.4, T 2.4.5; 
Durell, P 2.4.6) but an authoring aid is not available. Neither techniques, 
procedures, nor authoring aids are available for Block II. 3 -De scribe Entry 
Behavior. 
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During Phase III the learning objectives are classified so that the 
optimal instructional events to produce learning can be identified. A 
plan is made which specifies the allocation of training personnel, in- 
structional materials and plant requirements for the conduct of instruc- 
tion. At this time a determination of the most appropriate instructional 
delivsry system (considering a variety of media) is made. A review of 
instructional materials is made to determine whether* existing instruc- 
tion meets the requirements of the learning objectives. New instruction 
is developed for objectives not taught by existing materials. A pilot 
test is then conducted to try out the instruction. 

Phase III is the best developed in terms of the availability' of, 
authoring aids for print and audio-visual materials. At least one aid 
is available for each block. Authoring aids are needed however for the 
development of computer-based instructional materials. A considerable 
number of procedures exist for media selection and the development of 
instruction. 
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The plan which specifies the trainees, learning activities, and 
physical resources is initiated during Phase IV. The instructor re- 
ceives an orientation to the course, conducts the instruction, and 
records student data. 



Authoring aids are unavailable for both blocks of Phase IV. Pro- 
cedures were also unavailable for elock IV. 1. The single procedure 
identified for Block IV. 2 couldbe/ converted to an authoring aid? 
however, its scope is limiTieTr-tocomputer -based instruction. 
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During Phase V the data collected about student performance are 
evaluated to reveal deficiencies in the instruction. On-the-job perf- 
ormance of trainees is also of interest to determine whether the in- 
struction meets the needs of the Command* Finally the internal and 
external evaluation data are summarised to provide input, for revisions 
to be male on the completed instructional system. 

Authoring aids are unavailable for the evaluation phase* A variety 
of procedures exist for conducting internal and external evaluations but 
none use the activities and terminology specified in the ISD model. The 
procedure for revising instruction devised by Ellis et al, (P 5.3*1) will 
evolve into an authoring aid directly applicable to Block V*3, 



Summary and Plans for Research and Development 

In Figure 2 the availability of authoring aids for the complete 
ISD model is summarized* Briefly, ah authoring aid is needed for 
Block I*1-Analyze Job* Procedures exist for jpb analysis so an author- 
ing aid could be readily developed. Block 11*3 would require a greater 
effort because procedures would need to be developed as well. Phase III- 
UEVELOP has a sufficient number of authoring aids and procedures to 
recommend no further effort for print materials. However, authors aids 
are required for computer-based instructional materials* Phases IV- 
IMPLEMENT and V-CONTROL need authoring aid development* Phase V would 
, be a higher priority than Phase IV because quality control is essential 
to an instructional design effort. * 

Three concluding comments regarding this state-of-the-art assessment 
of Instructional Systems Development are salient* First, there exists a 
considerable lack of information on the best ways of managing the instruc- 
tional systems development process* Second, the interrelationship between 
the ISD process and instructional strategies is an important one, yet the 
knowledge and technology base for understanding this relationship is lack- 
ing* An ongoing state-of-the-art assessment in instructional strategies 
will address this issue* Third, although most persons in the field believe 
that the instructional system development process is an expensive one, there 
is a lack of accurate detailed ISD cost data* 
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The final product of this research and development effort is 
viewed as an integrated instructional system composed of a series 
of field tested self-paced modules. The modules will be ISD authoring 
aids, an author management system, and a computer-based instruction 
course to teach how to use both the job aids and management system* 
The curriculum will be designed so that extensive resources, instructor 
personnel or facility support will not be required. The modules will be 
available in the form of audio-visual materials, books, and documented 
computer programs. Evaluation data regarding its cost-effectiveness 
will also be available. 



